AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

Transformer-based autoregressive (AR) methods have achieved appealing performance for varied sequence-to-sequence generation tasks, e.g., neural machine translation, summarization, and code generation, but suffer from low inference efficiency. To speed up the stage, many non-autoregressive (NAR) strategies been proposed in past few years. Among them, conditional masked language model (CMLM) is one of most versatile frameworks, as it can support different sequence scenarios achieve very competitive on these tasks. In this paper, we further introduce a simple yet effective adaptive masking over strategy to enhance refinement capability decoder make encoder optimization easier. Experiments 3 tasks (neural generation) with 15 datasets total confirm that our method achieves significant improvement strong CMLM model. Surprisingly, yields state-of-the-art translation (34.62 BLEU WMT16 EN RO, 34.82 RO EN, 34.84 IWSLT De En) even better than AR Transformer 7 benchmark at least 2.2x speedup. Our available GitHub.